<!DOCTYPE html>
<!--

	Modified template for STM32CubeMX.AI purpose

	d0.1: 	jean-michel.delorme@st.com
			add ST logo and ST footer

	d2.0: 	jean-michel.delorme@st.com
			add sidenav support

	d2.1: 	jean-michel.delorme@st.com
			clean-up + optional ai_logo/ai meta data
			
==============================================================================
           "GitHub HTML5 Pandoc Template" v2.1 — by Tristano Ajmone           
==============================================================================
Copyright © Tristano Ajmone, 2017, MIT License (MIT). Project's home:

- https://github.com/tajmone/pandoc-goodies

The CSS in this template reuses source code taken from the following projects:

- GitHub Markdown CSS: Copyright © Sindre Sorhus, MIT License (MIT):
  https://github.com/sindresorhus/github-markdown-css

- Primer CSS: Copyright © 2016-2017 GitHub Inc., MIT License (MIT):
  http://primercss.io/

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The MIT License 

Copyright (c) Tristano Ajmone, 2017 (github.com/tajmone/pandoc-goodies)
Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
Copyright (c) 2017 GitHub Inc.

"GitHub Pandoc HTML5 Template" is Copyright (c) Tristano Ajmone, 2017, released
under the MIT License (MIT); it contains readaptations of substantial portions
of the following third party softwares:

(1) "GitHub Markdown CSS", Copyright (c) Sindre Sorhus, MIT License (MIT).
(2) "Primer CSS", Copyright (c) 2016 GitHub Inc., MIT License (MIT).

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
==============================================================================-->
<html>
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <meta name="keywords" content="STM32CubeMX, X-CUBE-AI, Neural Network, Validation, Quantization support, CLI, Code Generator, Automatic NN mapping tools" />
  <title>Evaluation report and metrics</title>
  <style type="text/css">
.markdown-body{
	-ms-text-size-adjust:100%;
	-webkit-text-size-adjust:100%;
	color:#24292e;
	font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";
	font-size:16px;
	line-height:1.5;
	word-wrap:break-word;
	box-sizing:border-box;
	min-width:200px;
	max-width:980px;
	margin:0 auto;
	padding:45px;
	}
.markdown-body a{
	color:#0366d6;
	background-color:transparent;
	text-decoration:none;
	-webkit-text-decoration-skip:objects}
.markdown-body a:active,.markdown-body a:hover{
	outline-width:0}
.markdown-body a:hover{
	text-decoration:underline}
.markdown-body a:not([href]){
	color:inherit;text-decoration:none}
.markdown-body strong{font-weight:600}
.markdown-body h1,.markdown-body h2,.markdown-body h3,.markdown-body h4,.markdown-body h5,.markdown-body h6{
	margin-top:24px;
	margin-bottom:16px;
	font-weight:600;
	line-height:1.25}
.markdown-body h1{
	font-size:2em;
	margin:.67em 0;
	padding-bottom:.3em;
	border-bottom:1px solid #eaecef}
.markdown-body h2{
	padding-bottom:.3em;
	font-size:1.5em;
	border-bottom:1px solid #eaecef}
.markdown-body h3{font-size:1.25em}
.markdown-body h4{font-size:1em}
.markdown-body h5{font-size:.875em}
.markdown-body h6{font-size:.85em;color:#6a737d}
.markdown-body img{border-style:none}
.markdown-body svg:not(:root){
	overflow:hidden}
.markdown-body hr{
	box-sizing:content-box;
	height:.25em;
	margin:24px 0;
	padding:0;
	overflow:hidden;
	background-color:#e1e4e8;
	border:0}
.markdown-body hr::before{display:table;content:""}
.markdown-body hr::after{display:table;clear:both;content:""}
.markdown-body input{margin:0;overflow:visible;font:inherit;font-family:inherit;font-size:inherit;line-height:inherit}
.markdown-body [type=checkbox]{box-sizing:border-box;padding:0}
.markdown-body *{box-sizing:border-box}.markdown-body blockquote{margin:0}
.markdown-body ol,.markdown-body ul{padding-left:2em}
.markdown-body ol ol,.markdown-body ul ol{list-style-type:lower-roman}
.markdown-body ol ol,.markdown-body ol ul,.markdown-body ul ol,.markdown-body ul ul{margin-top:0;margin-bottom:0}
.markdown-body ol ol ol,.markdown-body ol ul ol,.markdown-body ul ol ol,.markdown-body ul ul ol{list-style-type:lower-alpha}
.markdown-body li>p{margin-top:16px}
.markdown-body li+li{margin-top:.25em}
.markdown-body dd{margin-left:0}
.markdown-body dl{padding:0}
.markdown-body dl dt{padding:0;margin-top:16px;font-size:1em;font-style:italic;font-weight:600}
.markdown-body dl dd{padding:0 16px;margin-bottom:16px}
.markdown-body code{font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace}
.markdown-body pre{font:12px SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;word-wrap:normal}
.markdown-body blockquote,.markdown-body dl,.markdown-body ol,.markdown-body p,.markdown-body pre,.markdown-body table,.markdown-body ul{margin-top:0;margin-bottom:16px}
.markdown-body blockquote{padding:0 1em;color:#6a737d;border-left:.25em solid #dfe2e5}
.markdown-body blockquote>:first-child{margin-top:0}
.markdown-body blockquote>:last-child{margin-bottom:0}
.markdown-body table{display:block;width:100%;overflow:auto;border-spacing:0;border-collapse:collapse}
.markdown-body table th{font-weight:600}
.markdown-body table td,.markdown-body table th{padding:6px 13px;border:1px solid #dfe2e5}
.markdown-body table tr{background-color:#fff;border-top:1px solid #c6cbd1}
.markdown-body table tr:nth-child(2n){background-color:#f6f8fa}
.markdown-body img{max-width:100%;box-sizing:content-box;background-color:#fff}
.markdown-body code{padding:.2em 0;margin:0;font-size:85%;background-color:rgba(27,31,35,.05);border-radius:3px}
.markdown-body code::after,.markdown-body code::before{letter-spacing:-.2em;content:"\00a0"}
.markdown-body pre>code{padding:0;margin:0;font-size:100%;word-break:normal;white-space:pre;background:0 0;border:0}
.markdown-body .highlight{margin-bottom:16px}
.markdown-body .highlight pre{margin-bottom:0;word-break:normal}
.markdown-body .highlight pre,.markdown-body pre{padding:16px;overflow:auto;font-size:85%;line-height:1.45;background-color:#f6f8fa;border-radius:3px}
.markdown-body pre code{display:inline;max-width:auto;padding:0;margin:0;overflow:visible;line-height:inherit;word-wrap:normal;background-color:transparent;border:0}
.markdown-body pre code::after,.markdown-body pre code::before{content:normal}
.markdown-body .full-commit .btn-outline:not(:disabled):hover{color:#005cc5;border-color:#005cc5}
.markdown-body kbd{box-shadow:inset 0 -1px 0 #959da5;display:inline-block;padding:3px 5px;font:11px/10px SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;color:#444d56;vertical-align:middle;background-color:#fcfcfc;border:1px solid #c6cbd1;border-bottom-color:#959da5;border-radius:3px;box-shadow:inset 0 -1px 0 #959da5}
.markdown-body :checked+.radio-label{position:relative;z-index:1;border-color:#0366d6}
.markdown-body .task-list-item{list-style-type:none}
.markdown-body .task-list-item+.task-list-item{margin-top:3px}
.markdown-body .task-list-item input{margin:0 .2em .25em -1.6em;vertical-align:middle}
.markdown-body::before{display:table;content:""}
.markdown-body::after{display:table;clear:both;content:""}
.markdown-body>:first-child{margin-top:0!important}
.markdown-body>:last-child{margin-bottom:0!important}
.Alert,.Error,.Note,.Success,.Warning{padding:11px;margin-bottom:24px;border-style:solid;border-width:1px;border-radius:4px}
.Alert p,.Error p,.Note p,.Success p,.Warning p{margin-top:0}
.Alert p:last-child,.Error p:last-child,.Note p:last-child,.Success p:last-child,.Warning p:last-child{margin-bottom:0}
.Alert{color:#246;background-color:#e2eef9;border-color:#bac6d3}
.Warning{color:#4c4a42;background-color:#fff9ea;border-color:#dfd8c2}
.Error{color:#911;background-color:#fcdede;border-color:#d2b2b2}
.Success{color:#22662c;background-color:#e2f9e5;border-color:#bad3be}
.Note{color:#2f363d;background-color:#f6f8fa;border-color:#d5d8da}
.Alert h1,.Alert h2,.Alert h3,.Alert h4,.Alert h5,.Alert h6{color:#246;margin-bottom:0}
.Warning h1,.Warning h2,.Warning h3,.Warning h4,.Warning h5,.Warning h6{color:#4c4a42;margin-bottom:0}
.Error h1,.Error h2,.Error h3,.Error h4,.Error h5,.Error h6{color:#911;margin-bottom:0}
.Success h1,.Success h2,.Success h3,.Success h4,.Success h5,.Success h6{color:#22662c;margin-bottom:0}
.Note h1,.Note h2,.Note h3,.Note h4,.Note h5,.Note h6{color:#2f363d;margin-bottom:0}
.Alert h1:first-child,.Alert h2:first-child,.Alert h3:first-child,.Alert h4:first-child,.Alert h5:first-child,.Alert h6:first-child,.Error h1:first-child,.Error h2:first-child,.Error h3:first-child,.Error h4:first-child,.Error h5:first-child,.Error h6:first-child,.Note h1:first-child,.Note h2:first-child,.Note h3:first-child,.Note h4:first-child,.Note h5:first-child,.Note h6:first-child,.Success h1:first-child,.Success h2:first-child,.Success h3:first-child,.Success h4:first-child,.Success h5:first-child,.Success h6:first-child,.Warning h1:first-child,.Warning h2:first-child,.Warning h3:first-child,.Warning h4:first-child,.Warning h5:first-child,.Warning h6:first-child{margin-top:0}
h1.title,p.subtitle{text-align:center}
h1.title.followed-by-subtitle{margin-bottom:0}
p.subtitle{font-size:1.5em;font-weight:600;line-height:1.25;margin-top:0;margin-bottom:16px;padding-bottom:.3em}
div.line-block{white-space:pre-line}
  </style>
  <style type="text/css">code{white-space: pre;}</style>
  <style type="text/css">
	code.sourceCode > span { display: inline-block; line-height: 1.25; }
code.sourceCode > span { color: inherit; text-decoration: inherit; }
code.sourceCode > span:empty { height: 1.2em; }
.sourceCode { overflow: visible; }
code.sourceCode { white-space: pre; position: relative; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
code.sourceCode { white-space: pre-wrap; }
code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
}
pre.numberSource code
  { counter-reset: source-line 0; }
pre.numberSource code > span
  { position: relative; left: -4em; counter-increment: source-line; }
pre.numberSource code > span > a:first-child::before
  { content: counter(source-line);
    position: relative; left: -1em; text-align: right; vertical-align: baseline;
    border: none; display: inline-block;
    -webkit-touch-callout: none; -webkit-user-select: none;
    -khtml-user-select: none; -moz-user-select: none;
    -ms-user-select: none; user-select: none;
    padding: 0 4px; width: 4em;
    color: #aaaaaa;
  }
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa;  padding-left: 4px; }
div.sourceCode
  {   }
@media screen {
code.sourceCode > span > a:first-child::before { text-decoration: underline; }
}
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code span.at { color: #7d9029; } /* Attribute */
code span.bn { color: #40a070; } /* BaseN */
code span.bu { } /* BuiltIn */
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code span.ch { color: #4070a0; } /* Char */
code span.cn { color: #880000; } /* Constant */
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
code span.dt { color: #902000; } /* DataType */
code span.dv { color: #40a070; } /* DecVal */
code span.er { color: #ff0000; font-weight: bold; } /* Error */
code span.ex { } /* Extension */
code span.fl { color: #40a070; } /* Float */
code span.fu { color: #06287e; } /* Function */
code span.im { } /* Import */
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
code span.op { color: #666666; } /* Operator */
code span.ot { color: #007020; } /* Other */
code span.pp { color: #bc7a00; } /* Preprocessor */
code span.sc { color: #4070a0; } /* SpecialChar */
code span.ss { color: #bb6688; } /* SpecialString */
code span.st { color: #4070a0; } /* String */
code span.va { color: #19177c; } /* Variable */
code span.vs { color: #4070a0; } /* VerbatimString */
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
  </style>
  <style type="text/css">:root { --main-hx-color: rgb(0,32,88); --sidenav-font-size: 90%;}html {}* {xbox-sizing: border-box;}.st_header h1.title,.st_header p.subtitle {text-align: left;}.st_header h1.title {color: var(--main-hx-color)}.st_header p.subtitle {color: var(--main-hx-color)}.st_header h1.title.followed-by-subtitle {margin-bottom:5px;}.st_header p.revision {display: inline-block;width:70%;}.st_header div.author {font-style: italic;}.st_header div.summary {border-top: solid 1px #C0C0C0;background: #ECECEC;padding: 5px;}.st_footer img {float: right;}.markdown-body #header-section-number {font-size:120%;}.markdown-body h1 {border-bottom:1px solid #74767a;padding-bottom: 2px;padding-top: 10px;}.markdown-body h2 {padding-bottom: 5px;padding-top: 10px;}.markdown-body h2 code {background-color: rgb(255, 255, 255);}#func.sourceCode {border-left-style: solid;border-color: rgb(0, 32, 82);border-color: rgb(255, 244, 191);border-width: 8px;padding:0px;}pre > code {border: solid 1px blue;font-size:60%;}codeXX {border: solid 1px blue;font-size:60%;}#func.sourceXXCode::before {content: "Synopsis";padding-left:10px;font-weight: bold;}figure {padding:0px;margin-left:5px;margin-right:5px;margin-left: auto;margin-right: auto;}img[data-property="center"] {display: block;margin-top: 10px;margin-left: auto;margin-right: auto;padding: 10px;}figcaption {text-align:left;  border-top: 1px dotted #888;padding-bottom: 20px;margin-top: 10px;}section.st_footer {font-size:80%;}div.stnotice {width:80%;}h1 code, h2 code {font-size:120%;}	.markdown-body table {width: 100%;margin-left:auto;margin-right:auto;}.markdown-body img {border-radius: 4px;padding: 5px;display: block;margin-left: auto;margin-right: auto;width: auto;}.markdown-body .st_header img, .markdown-body {border: none;border-radius: none;padding: 5px;display: block;margin-left: auto;margin-right: auto;width: auto;box-shadow: none;}.markdown-body {margin: 10px;padding: 10px;width: auto;font-family: "Arial", sans-serif;color: #03234B;}.markdown-body h1, .markdown-body h2, .markdown-body h3 {   color: var(--main-hx-color)}.markdown-body:hover {}.markdown-body .contents {}.markdown-body .toc-title {}.markdown-body .contents li {list-style-type: none;}.markdown-body .contents ul {padding-left: 10px;}.markdown-body .contents a {color: #3CB4E6; }.sidenav {font-family: "Arial", sans-serif;font-family: segoe ui, verdona;color: #3CB4E6; color: #03234B; color: var(--main-hx-color);height: 100%;position: fixed;z-index: 1;top: 0;left: 0;margin-right: 10px;margin-left: 10px; overflow-x: hidden;}hr.new1 {border-width: thin;border-top: 1px solid #3CB4E6; margin-right: 10px;margin-top: -10px;}.sidenav #sidenav_header {margin-top: 10px;border: 1px;}.sidenav #sidenav_header img {float: left;}.sidenav #sidenav_header a {margin-left: 0px;margin-right: 0px;padding-left: 0px;color: #3CB4E6; color: #03234B; color: var(--main-hx-color)}.sidenav #sidenav_header a:hover {background-size: auto;color: #FFD200; }.sidenav #sidenav_header a:active {  }.sidenav > ul {background-color: rgba(57, 169, 220, 0.05);border-radius: 10px;padding-bottom: 10px;padding-top: 10px;padding-right: 10px;margin-right: 10px;}.sidenav a {padding: 2px 2px;text-decoration: none;font-size: var(--sidenav-font-size);  display:table;}.sidenav > ul > li,.sidenav > ul > li > ul > li { padding-right: 5px;padding-left: 5px;}.sidenav > ul > li > a { color: #03234B;  color: var(--main-hx-color)}.sidenav > ul > li > ul > li > a { color: #03234B; color: #3CB4E6; color: #03234B; font-weight: lighter;padding-left: 10px;}.sidenav > ul > li > ul > li > ul > li > a { display: None;}.sidenav li {list-style-type: none;}.sidenav ul {padding-left: 0px;}.sidenav > ul > li > a:hover,.sidenav > ul > li > ul > li > a:hover {background-color: rgba(70, 70, 80, 0.1); background-clip: border-box;margin-left: -10px;padding-left: 10px;}.sidenav > ul > li > a:hover {padding-right: 15px;width: 230px;	}.sidenav > ul > li > ul > li > a:hover {padding-right: 10px;width: 230px;	}.sidenav > ul > li > a:active { color: #FFD200; }.sidenav code {}.sidenav {width: 280px;}#sidenav {margin-left: 300px;display:block;}.markdown-body .print-contents {visibility:hidden;}.markdown-body .print-toc-title {visibility:hidden;}.markdown-body {max-width: 980px;min-width: 200px;padding: 40px;border-style: solid;border-style: outset;border-color: rgba(104, 167, 238, 0.089);border-radius: 5px;}@media screen and (max-height: 450px) {.sidenav {padding-top: 15px;}.sidenav a {font-size: 18px;}#sidenav {margin-left: 10px; }.sidenav {visibility:hidden;}.markdown-body {margin: 10px;padding: 40px;width: auto;border: 0px;}}@media screen and (max-width: 1024px) {.sidenav {visibility:hidden;}.markdown-body {margin: 10px;padding: 40px;width: auto;border: 0px;}#sidenav {margin-left: 10px;}}@media print {.sidenav {visibility:hidden;}#sidenav {margin-left: 10px;}.markdown-body {margin: 10px;padding: 10px;width:auto;border: 0px;}@page {size: A4;  margin:2cm;padding:2cm;margin-top: 1cm;padding-bottom: 1cm;}* {xbox-sizing: border-box;font-size:90%;}a {font-size: 100%;color: yellow;}.markdown-body article {xbox-sizing: border-box;font-size:100%;}.markdown-body p {windows: 2;orphans: 2;}.pagebreakerafter {page-break-after: always;padding-top:10mm;}.pagebreakbefore {page-break-before: always;}h1, h2, h3, h4 {page-break-after: avoid;}div, code, blockquote, li, span, table, figure {page-break-inside: avoid;}}</style>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->




<link href="" rel="shortcut icon">

</head>



<body>

		<div class="sidenav">
		<div id="sidenav_header">
							<img src="" title="STM32CubeMX.AI logo" align="left" height="70" />
										<br />5.2.0<br />
										<a href="#doc_title"> Evaluation report and metrics </a>
					</div>
		<div id="sidenav_header_button">
			 
							<ul>
					<li><p><a id="index" href="index.html">[ Index ]</a></p></li>
				</ul>
						<hr class="new1">
		</div>	

		<ul>
<li><a href="#introduction">Introduction</a><ul>
<li><a href="#metrics">Metrics</a></li>
<li><a href="#classifier-and-regressor-models">Classifier and regressor models</a></li>
<li><a href="#ref_valio_arg">Input validation files</a></li>
<li><a href="#ref_post_proc_support">Output validation files for post-processing support</a></li>
</ul></li>
<li><a href="#metrics-1">Metrics</a><ul>
<li><a href="#ref_complexity">Computational complexity: MACC and cycles/MACC</a></li>
<li><a href="#ref_memory_occupancy">Memory-related metrics</a></li>
<li><a href="#ref_acc">Classification accuracy (acc)</a></li>
<li><a href="#ref_mae">Mean Absolute Error (mae)</a></li>
<li><a href="#ref_rmse">Root Mean Square Error (rmse)</a></li>
<li><a href="#ref_l2r">L2 relative error (l2r)</a></li>
<li><a href="#ref_cm">Confusion matrix (CM)</a></li>
</ul></li>
<li><a href="#use-cases">Use cases</a><ul>
<li><a href="#ref_f_inputs">32b float model - Input samples only</a></li>
<li><a href="#ref_f_io">32b float model - Input/output samples</a></li>
<li><a href="#ref_q_inputs">Quantized model - Input samples only</a></li>
<li><a href="#tflite-quantized-model---inputoutput-samples">TFLite Quantized model - Input/output samples</a></li>
<li><a href="#model-with-multiple-io">Model with multiple IO</a></li>
</ul></li>
<li><a href="#ref_script_ex">Post-processing example</a></li>
<li><a href="#references">References</a></li>
<li><a href="#revision-history">Revision history</a></li>
</ul>
	</div>
	<article id="sidenav" class="markdown-body">
	


<header>
<section class="st_header" id="doc_title">

<div class="himage">
	<img src="" title="STM32CubeMX.AI" align="right" height="70" />
	<img src="" title="STM32" align="right" height="90" />
</div>

<h1 class="title followed-by-subtitle">Evaluation report and metrics</h1>

	<p class="subtitle">X-CUBE-AI Expansion Package</p>

	<div class="revision">r1.4</div>

	<div class="ai_platform">
		AI PLATFORM r5.2.0
					(Embedded Inference Client API 1.1.0)
			</div>
			Command Line Interface r1.4.0
	




</section>
</header>




<section id="introduction" class="level1">
<h1>Introduction</h1>
<p>This article describes the different metrics which are computed and used to test and to evaluate the generated C-files (or C-model).</p>
<section id="metrics" class="level2">
<h2>Metrics</h2>
<table>
<colgroup>
<col style="width: 14%"></col>
<col style="width: 13%"></col>
<col style="width: 39%"></col>
<col style="width: 32%"></col>
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">metric</th>
<th style="text-align: left;">category</th>
<th style="text-align: left;">description</th>
<th style="text-align: left;">model type, format</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;"><code>MACC</code></td>
<td style="text-align: left;">perf</td>
<td style="text-align: left;"><a href="#ref_complexity">computational complexity</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="even">
<td style="text-align: left;"><code>ROM/RAM</code></td>
<td style="text-align: left;">memory</td>
<td style="text-align: left;"><a href="#ref_memory_occupancy">memory-related metrics</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="odd">
<td style="text-align: left;"><code>ACC</code></td>
<td style="text-align: left;">perf/error</td>
<td style="text-align: left;"><a href="#ref_acc">accuracy (Classification accuracy)</a></td>
<td style="text-align: left;">only classifier models (float and integer format)</td>
</tr>
<tr class="even">
<td style="text-align: left;"><code>RMSE</code></td>
<td style="text-align: left;">perf/error</td>
<td style="text-align: left;"><a href="#ref_rmse">Root Mean Square Error</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="odd">
<td style="text-align: left;"><code>MAE</code></td>
<td style="text-align: left;">perf/error</td>
<td style="text-align: left;"><a href="#ref_mae">Mean Absolute Error</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="even">
<td style="text-align: left;"><code>L2r</code></td>
<td style="text-align: left;">perf/error</td>
<td style="text-align: left;"><a href="#ref_l2r">L2 relative Error</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="odd">
<td style="text-align: left;"><code>CM</code></td>
<td style="text-align: left;">perf/error</td>
<td style="text-align: left;"><a href="#ref_cm">Confusion Matrix</a></td>
<td style="text-align: left;">only classifier models (float and integer format)</td>
</tr>
</tbody>
</table>
<p>Evaluation of the perf/error metrics are computed with the following data:</p>
<ul>
<li><code>[I]</code> designates the list of the input samples which are used to feed the original model and the C-model. It can be provided by the user (see <a href="#ref_valio_arg">“inputs validation files”</a>) or randomly generated.</li>
<li><code>[P]</code> designates the list of the predicted samples inferred by the C-model.</li>
<li><code>[R&#39;]</code> designates the list of the predicted samples inferred by the original model.</li>
<li>{optional} <code>[R]</code> designates, the list of the predicted output samples provided by the user which are used as ground truth or reference values.</li>
</ul>
<div id="fig:id_m_compute" class="fignos">
<figure>
<img src="" property="center" style="width:85.0%" alt /><figcaption><span>Figure 1:</span> Metrics evaluation data flow</figcaption>
</figure>
</div>
<p>Results are summarized in a simple table.</p>
<pre><code>Evaluation report (summary)
--------------------------------------------------------------------------------------------------------
Mode                 acc      rmse      mae       l2r       tensor
--------------------------------------------------------------------------------------------------------
x86 C-model #1       92.68%   0.053623  0.005785  0.340042  dense_4_nl [ai_float, (1, 1, 36), m_id=10]
original model #1    92.68%   0.053623  0.005785  0.340042  dense_4_nl [ai_float, (1, 1, 36), m_id=10]
X-cross #1           100.00%  0.000000  0.000000  0.000000  dense_4_nl [ai_float, (1, 1, 36), m_id=10]

L2r error : 1.11012369e-07 (expected to be &lt; 0.01)
</code></pre>
<ul>
<li><code>&#39;X-cross&#39;</code> line indicates the metrics which are evaluated with the <code>[P]</code> and <code>[R&#39;]</code> values. The predicted values <code>[R&#39;]</code> from the original model are considered as the <strong>reference</strong> values. If <code>[R]</code> is not provided, only <code>&#39;X-cross&#39;</code> metrics are computed else all metrics are reported for the <code>&#39;C-model&#39;</code> and <code>&#39;original model&#39;</code>. <code>&#39;C-model&#39;</code> refers to the execution of the <em>X86</em> or <em>STM32</em> generated C-model. In both case, the same validation flow is applied (refer to <a href="https://www.st.com/resource/en/user_manual/dm00570145.pdf">[2]</a>).</li>
</ul>
<div class="Warning">
<p><strong>Note</strong> — All input data and generated output data are always saved (without modification) in the particular files, see <a href="#ref_post_proc_support">“Output validation files for post-processing support”</a> section.</p>
</div>
<div class="Warning">
<p><strong>Note</strong> — Evaluation of a generated C-model is more accurate with the custom representative input values. Because the generated random samples are uniformly distributed with a fixed seed inside <code>&#39;[-1.0, 1.0]&#39;</code> range, and they are not necessarily representative of the input space of data used to train the model.</p>
</div>
</section>
<section id="classifier-and-regressor-models" class="level2">
<h2>Classifier and regressor models</h2>
<p>In the case where a regressor model is considered by the validation engine, only the <code>&#39;ACC&#39;</code> and <code>&#39;CM&#39;</code> metrics are NOT computed, all other metrics are evaluated. According the predicted values (<code>[R&#39;]</code> or <code>[R]</code>), the type of the model is automatically detected. User has the possibility to force the computation of the <code>ACC</code>/<code>CM</code> metrics with the <code>&#39;--classifier&#39;</code> flag in the command line.</p>
<pre><code>Evaluation report (summary)
--------------------------------------------------
Mode                acc       rmse      mae
--------------------------------------------------
x86 C-model         n.a.      0.020375  0.011562
original model      n.a.      0.020461  0.011608
X-cross             n.a.      0.000594  0.000292

L2r error : 2.45757354e-07 (expected to be &lt; 0.01)
</code></pre>
</section>
<section id="ref_valio_arg" class="level2">
<h2>Input validation files</h2>
<p>The user can provide the inputs and associated ground truth or reference values in a simple file (<code>&#39;npz&#39;</code> numpy format) or separated files (<code>&#39;npy&#39;</code> numpy format or <code>&#39;csv&#39;</code> format).</p>
<ul>
<li><code>&#39;csv&#39;</code> files are the normal text files with a flattened version of the input (or output) tensors, one by line. Comma <code>&#39;,&#39;</code> separator is used to separate the values.</li>
<li><code>&#39;npy&#39;</code> files are the standard numpy binary format to store a single array.</li>
<li><code>&#39;npz&#39;</code> file is a standard numpy binary format to store several arrays. The following dict entries (or keys) are supported to store the data: <code>&#39;x_test&#39;</code> and <code>&#39;y_test&#39;</code> or <code>&#39;in_0&#39;</code> and <code>&#39;out_0&#39;</code> or <code>&#39;m_inputs&#39;</code> and <code>&#39;m_outputs&#39;</code> or <code>&#39;m_inputs_1&#39;</code> and <code>&#39;m_outputs_1&#39;</code>. For network with multiple IO, <code>&#39;m_inputs_&lt;idx&gt;&#39;</code> and <code>&#39;m_outputs_&lt;idx&gt;&#39;</code> keys with <code>idx</code> starting with <code>1</code> should be used.</li>
</ul>
<p>32b float and int8/uint8 numbers are supported. For the csv files, a particular tag (<code>&#39;dtype=uint8&#39;</code> or <code>&#39;dtype=int8&#39;</code>) must be defined in the five first comment lines to indicate the type of the data, else 32b float number will be used.</p>
<pre class="dosbatch"><code># Example of csv file (32-b float number)
# comment line
-1.076007485389709473e+00,6.278980255126953125e+00,.. 3.949900865554809570e+00
1.160453605651855469e+01,1.707991600036621094e+01,.. 1.334794044494628906e+00
...</code></pre>
<pre class="dosbatch"><code># Example of csv file (uint8 number)
# dtype=uint8
50, 65, 71, 71
4.800000000000000000e+01,6.700000000000000000e+01,7.300000000000000000e+01,6.700000000000000000e+01
...</code></pre>
<ul>
<li><p>Lines from the <code>&#39;csv&#39;</code> file are always parsed as the 32b float numbers, after they are converted to int8/uint8 type if requested.</p></li>
<li><p>For the quantized models, if a 32b float data set is used, a pre-process is applied to convert the data according the expected format. Same conversion is done for the outputs. Warning if <code>&#39;uint8&#39;</code> data are provided, and the requested quantized model expects <code>&#39;int8&#39;</code> or <code>&#39;float32&#39;</code> type an exception will be raised.</p></li>
<li><p>For network with multiple IO, one <code>csv</code> file should be provided by input (respectively by output). The order in the CLI is used to know the index of the network inputs/outputs (name of the file is not considered). For the <code>npz</code> file, the index <code>&lt;idx&gt;</code> of the key will be used.</p>
<pre class="dosbatch"><code>$ stm32ai validate &lt;model_file&gt; -vi inputs.csv inputs_extra.csv -vo outputs.csv</code></pre></li>
<li><p>For a classifier, ground truth values should be provided as <em>one-shot-encoding</em> format to match with the network output shape.</p></li>
<li><p>When the user data set is loaded, shape and types is reported in the log.</p>
<pre class="dosbatch"><code>-- Setting inputs (and outputs) data  
Using input file(s), shapes=[(10, 99)] dtype=[float32]
Using reference output file(s), shapes=[(10, 5)] dtype=[float32]
-- Setting inputs (and outputs) data - done (elapsed time 0.011s)</code></pre></li>
<li><p>Generated output validation files from a previous “validate” command can be used as input files.</p></li>
</ul>
</section>
<section id="ref_post_proc_support" class="level2">
<h2>Output validation files for post-processing support</h2>
<p>The inputs and generated predicted values are <strong>always</strong> saved in different files allowing to apply a <a href="#ref_script_ex">post-processing script</a> to evaluate user-defined metrics. Data are always saved according the IO data type, in HWC order or channel-last format. For network with multiple IO, there is one <code>&#39;csv&#39;</code> file by input (respectively output). Name of the file is suffixed with the respective index.</p>
<pre class="dosbatch"><code>...
Saving data in &quot;&lt;output-directory-path&gt;\&quot; folder
 creating &quot;network_val_m_inputs_1.csv&quot;  dtype=[float32]
 creating &quot;network_val_c_inputs_1.csv&quot;  dtype=[float32]
 creating &quot;network_val_m_outputs_1.csv&quot;  dtype=[uint8]
 creating &quot;network_val_c_outputs_1.csv&quot;  dtype=[uint8]
 creating &quot;network_val_io.npz&quot;
...</code></pre>
<ul>
<li>The <code>csv</code> files are the normal text files with a flattened version of the input (or output) tensors, one by line. Comma <code>,</code> separator is used to separate the values.</li>
<li>The <code>npz</code> file is a standard numpy binary format to store several arrays. The following dict entries are respectively used to store the data: <code>&#39;m_inputs_&lt;idx&gt;&#39;</code>, <code>&#39;c_inputs_&lt;idx&gt;&#39;</code>, <code>&#39;m_outputs_&lt;idx&gt;&#39;</code> and <code>&#39;c_outputs_&lt;idx&gt;&#39;</code>.</li>
</ul>
<div class="Warning">
<p><strong>Limitation</strong> – For the <code>*.csv</code> files, only the <strong>first 64 samples</strong> are stored and if the number of item by sample is lower than <strong>1024</strong>. The <code>npz</code> file is always created.</p>
</div>
</section>
</section>
<section id="metrics-1" class="level1">
<h1>Metrics</h1>
<section id="ref_complexity" class="level2">
<h2>Computational complexity: MACC and cycles/MACC</h2>
<p>When a model is analyzed (refer to <a href="command_line_interface.html">[3], “Analyze command”</a> section or <a href="https://www.st.com/resource/en/user_manual/dm00570145.pdf">[2], “Dimensioning information report”</a> section), a global logical computational complexity is reported: <code>&#39;MACC&#39;</code>. It indicates the number of multiply-and-accumulate operations which are requested to perform an inference. Reported value is computed independently of the data format (floating-point, fixed-point or integer) or the underlying C-implementation or/and possible optimization.</p>
<p>As mentioned in <a href="https://www.st.com/resource/en/user_manual/dm00570145.pdf">[2]</a>, <em>aiSystemPerformance</em> and <em>aiValidation</em> applications permit to report the average number of clocks requested for the whole C-model allowing to compute the number of requested <em>cycles by MACC</em>. This indicator highlights the <em>global efficiency</em> of the underlying C-implementation (including the HW platform aspects).</p>
<figure>
<img src="" property="center" style="width:55.0%" alt />
</figure>
<pre class="dosbatch"><code>...
Results for 10 inference(s) @480/240MHz (macc:336084)
 device      : 0x450/STM32H743/753 and STM32H750 @480MHz/240MHz (FPU is present) lat=4 Core:I$/D$
 duration    : 4.522 ms (average)
 CPU cycles  : 2170476 (average)
 cycles/MACC : 6.46 (average for all layers)
...</code></pre>
</section>
<section id="ref_memory_occupancy" class="level2">
<h2>Memory-related metrics</h2>
<p>When a model is rendered (refer to <a href="command_line_interface.html">[3], “Analyze command”</a> section), two main memory-related metrics (<code>&#39;weights (ro)&#39;</code> or <code>&#39;ROM&#39;</code> and <code>&#39;activations (rw)&#39;</code> or <code>&#39;RAM&#39;</code>) are reported to know the size of the memory which is requested to integrate the generated C-files. Please refer to <a href="embedded_client_api.html">[5]</a>, <em>“AI buffers and privileged placement”</em> section to have more details about the integration aspects. Note that the generation of the relocatable binary file (refer to <a href="relocatable.html">[9]</a>) allows to have more details about the final requested memory layout. Requested size of the AI .bss/.data/.rodata/.text sections for the kernels and network C-structures are also reported.</p>
<pre class="dosbatch"><code>Runtime memory layout (series=&quot;stm32f4&quot;)
--------------------------------------------------------------------------------
section      size (bytes)
--------------------------------------------------------------------------------
header                100 *
txt                 7,864      network+kernel
rodata                128      network+kernel
data                1,756      network+kernel
bss                   132      network+kernel
got                   108 *
rel                   504 *
weights            15,560      network
--------------------------------------------------------------------------------
flash size         25,308 + 712 (+2.81%) *
ram size            1,888 + 108 (+5.72%) *
--------------------------------------------------------------------------------
bin size           26,024      binary image
act. size             192      activations buffer

(*) extra bytes for relocatable support</code></pre>
</section>
<section id="ref_acc" class="level2">
<h2>Classification accuracy (acc)</h2>
<p><em>Classification accuracy</em> is what we usually mean, when the term <em>accuracy</em> is used. <em>ACC</em> is the ratio between of correct predictions to the total number of inputs. This indicator evaluates the performance of the <em>classifier</em> model, if a <em>regressor</em> type is passed, the <em>ACC</em> is <strong>NOT</strong> calculated and <code>n.a.</code> value is reported.</p>
<figure>
<img src="" property="center" style="width:45.0%" alt />
</figure>
<div class="sourceCode" id="cb10"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb10-1"><a href="#cb10-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb10-2"><a href="#cb10-2"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> accuracy_score</span>
<span id="cb10-3"><a href="#cb10-3"></a></span>
<span id="cb10-4"><a href="#cb10-4"></a><span class="kw">def</span> acc(ref, pred):</span>
<span id="cb10-5"><a href="#cb10-5"></a>  <span class="co">&quot;&quot;&quot;Classification accuracy (ACC).&quot;&quot;&quot;</span></span>
<span id="cb10-6"><a href="#cb10-6"></a>  <span class="cf">return</span> accuracy_score(np.argmax(ref, axis<span class="op">=</span><span class="dv">1</span>), np.argmax(pred, axis<span class="op">=</span><span class="dv">1</span>))</span></code></pre></div>
</section>
<section id="ref_mae" class="level2">
<h2>Mean Absolute Error (mae)</h2>
<p><em>MAE</em> is the average of the difference between the original value (or reference value, Rj) and the predicted value Pj. It gives the measure of how far the predictions were from the actual output. However, they don’t gives any idea of the direction of the error i.e. whether we are under predicting the data or over predicting the data. <em>MAE</em> is computed for the flattened array (element-wise along the array), returning a scalar value. If [R] is not provided [R’] is used.</p>
<figure>
<img src="" property="center" style="width:30.0%" alt />
</figure>
<div class="sourceCode" id="cb11"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb11-1"><a href="#cb11-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb11-2"><a href="#cb11-2"></a></span>
<span id="cb11-3"><a href="#cb11-3"></a><span class="kw">def</span> mae(ref, pred):</span>
<span id="cb11-4"><a href="#cb11-4"></a>  <span class="co">&quot;&quot;&quot;Return Mean Absolute Error (MAE).&quot;&quot;&quot;</span></span>
<span id="cb11-5"><a href="#cb11-5"></a>  <span class="cf">return</span> (np.<span class="bu">abs</span>(ref <span class="op">-</span> pred).astype(np.float64)).mean()</span></code></pre></div>
</section>
<section id="ref_rmse" class="level2">
<h2>Root Mean Square Error (rmse)</h2>
<p><em>RMSE</em> is quite similar to <em>MAE</em>, the only difference being that <em>RMSE</em> takes the average of the square of the difference between the original values and the predicted values. As, we take square of the error, the effect of larger errors become more pronounced then smaller error, hence the model can now focus more on the larger errors. <em>RMSE</em> is computed for the flattened array (element-wise along the array), returning a scalar value. If [R] is not provided [R’] is used.</p>
<figure>
<img src="" property="center" style="width:30.0%" alt />
</figure>
<div class="sourceCode" id="cb12"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb12-1"><a href="#cb12-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb12-2"><a href="#cb12-2"></a></span>
<span id="cb12-3"><a href="#cb12-3"></a><span class="kw">def</span> rmse(ref, pred):</span>
<span id="cb12-4"><a href="#cb12-4"></a>  <span class="co">&quot;&quot;&quot;Return Root Mean Squared Error (RMSE).&quot;&quot;&quot;</span></span>
<span id="cb12-5"><a href="#cb12-5"></a>  <span class="cf">return</span> np.sqrt(((ref <span class="op">-</span> pred).astype(np.float64) <span class="op">**</span> <span class="dv">2</span>).mean())</span></code></pre></div>
</section>
<section id="ref_l2r" class="level2">
<h2>L2 relative error (l2r)</h2>
<p><em>L2r</em> is the scalar value of the relative 2-norm or Euclidean distance between the generated values of the original model [R’] and the C-model [P]. This metric is only used with a 32b float model. When possible, this metric is also reported for the output of a C-layer matching with the original layer (refer to <a href="command_line_interface.html">[3]</a>, <em>“L2r error report”</em> sub-section).</p>
<figure>
<img src="" property="center" style="width:35.0%" alt />
</figure>
<div class="sourceCode" id="cb13"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb13-1"><a href="#cb13-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb13-2"><a href="#cb13-2"></a></span>
<span id="cb13-3"><a href="#cb13-3"></a><span class="kw">def</span> l2r(ref, pred):</span>
<span id="cb13-4"><a href="#cb13-4"></a>  <span class="co">&quot;&quot;&quot;Compute L2 relative error&quot;&quot;&quot;</span></span>
<span id="cb13-5"><a href="#cb13-5"></a>  <span class="kw">def</span> magnitude(v):</span>
<span id="cb13-6"><a href="#cb13-6"></a>    <span class="cf">return</span> np.sqrt(np.<span class="bu">sum</span>(np.square(v).flatten()))</span>
<span id="cb13-7"><a href="#cb13-7"></a>  mag <span class="op">=</span> magnitude(pred) <span class="op">+</span> np.finfo(np.float32).eps</span>
<span id="cb13-8"><a href="#cb13-8"></a>  <span class="cf">return</span> magnitude(ref <span class="op">-</span> pred) <span class="op">/</span> mag</span></code></pre></div>
</section>
<section id="ref_cm" class="level2">
<h2>Confusion matrix (CM)</h2>
<p>When custom outputs are provided and the model is considered as a classifier, a confusion matrix is reported for the C-model and the reference model. It describes the complete performance of the model. Note that if a <em>regressor</em> type is passed, the confusion matrix is <strong>NOT</strong> calculated only the <a href="#ref_rmse">RMSE</a> and <a href="#ref_mae">MAE</a> metrics are reported.</p>
<pre class="dosbatch"><code>8 classes (50 samples)
------------------------------------------------
C0         4    .    .    .    .    .    .    .
C1         .    9    .    .    .    .    .    .
C2         .    .    6    .    .    .    .    .
C3         .    .    .    7    .    .    .    .
C4         .    .    .    .    5    .    .    .
C5         .    .    .    .    .    6    .    .
C6         .    .    .    .    .    .    7    .
C7         .    .    .    .    .    .    .    6</code></pre>
<p><strong><em>X-Cross</em></strong> confusion matrix or accuracy uses the outputs of the reference model to build the ground truth values. They are used to evaluate and to compare the performance of the C-model.</p>
<div class="Warning">
<p><strong>Limitation</strong> — The confusion matrix is only displayed when the number of class is lower or equal to 20.</p>
</div>
</section>
</section>
<section id="use-cases" class="level1">
<h1>Use cases</h1>
<section id="ref_f_inputs" class="level2">
<h2>32b float model - Input samples only</h2>
<p>Default case, where only the input values are used to evaluate the generated C-model implementing a 32b float model. The same inputs [I] (random data or custom data) are used to feed the original model and the generated C-model. The outputs [P] and [R’] are mainly used to build the <em>L2r</em> error in this case. However, [R’] is also used as reference to compute the X-cross metrics: <em>ACC</em>, <em>RMSE</em> and <em>MAE</em>. These metrics (and associated confusion matrix) allow to enhance the <em>L2r</em> metric with additional indicators, in particular when random data are used and compression factor is applied.</p>
<div id="fig:id_f_inputs" class="fignos">
<figure>
<img src="" property="center" style="width:85.0%" alt /><figcaption><span>Figure 1:</span> 32b float model with input samples only</figcaption>
</figure>
</div>
<p>Typical output during the execution of the <code>&#39;validate&#39;</code> command with random inputs.</p>
<pre class="dosbatch"><code>$ stm32ai validate -m &lt;modified_model_file&gt;.h5 --full
...
-- Setting inputs (and outputs) data
Using random inputs, shapes=[(10, 99)] dtype=[float32]
-- Running X86 C-model
-- Running X86 C-model - done (elapsed time 0.102s)
-- Running original model
-- Running original model - done (elapsed time 0.303s)

Saving data in &quot;&lt;output-directory-path&gt;&quot; folder
 creating &quot;network_val_m_inputs_1.csv&quot;  dtype=[float32]
 creating &quot;network_val_c_inputs_1.csv&quot;  dtype=[float32]
 creating &quot;network_val_m_outputs_1.csv&quot;  dtype=[float32]
 creating &quot;network_val_c_outputs_1.csv&quot;  dtype=[float32]
 creating &quot;network_val_io.npz&quot;

Cross accuracy report #1 (reference vs C-model)
----------------------------------------------------------------------------------------------------
NOTE: the output of the reference model is used as ground truth/reference value

 acc=100.00%, rmse=0.000000, mae=0.000000

 5 classes (10 samples)
 ---------------------------------
 C0        9    .    .    .    .
 C1        .    0    .    .    .
 C2        .    .    0    .    .
 C3        .    .    .    0    .
 C4        .    .    .    .    1


Evaluation report (summary)
------------------------------------------------------------
Mode                   acc       rmse      mae
------------------------------------------------------------
X-cross #1             100.00%   0.000000  0.000000

L2r error : 1.02767139e-07 (expected to be &lt; 0.01)

Creating report file &lt;output-directory-path&gt;\network_validate_report.txt

Complexity/l2r error per-layer - macc=3,960 rom=15,560
----------------------------------------------------------------------------------------------------
id  layer (type)                macc                    rom                      l2r error
----------------------------------------------------------------------------------------------------
0   dense_1 (Dense)             |||||||||||||||  82.5%  ||||||||||||||||  84.8%  1.22358045e-07
1   activation_1 (Nonlinearity) |                 0.8%  |                  0.0%  1.22744325e-07 *
2   dense_2 (Dense)             ||||             12.5%  ||||              13.1%  1.12652899e-07
3   activation_2 (Nonlinearity) |                 0.4%  |                  0.0%  9.49384997e-08
4   dense_3 (Dense)             |                 1.9%  |                  2.1%  9.90144144e-08
5   activation_3 (Nonlinearity) |                 1.9%  |                  0.0%  1.02767139e-07
----------------------------------------------------------------------------------------------------

...</code></pre>
</section>
<section id="ref_f_io" class="level2">
<h2>32b float model - Input/output samples</h2>
<p>This case is the recommended validation flow to evaluate the generated C-model implementing a 32b float model. As reference samples are provided, all metrics for each models (reference and c-model) are computed including X-cross values for quick evaluation. For a classifier model, the provided outputs (ground truth values) are used to report the <a href="#ref_acc">classification accuracy</a>.</p>
<div id="fig:id_f_io" class="fignos">
<figure>
<img src="" property="center" style="width:85.0%" alt /><figcaption><span>Figure 2:</span> Floating-point model with IO values</figcaption>
</figure>
</div>
<p>Typical output during the execution of the “validate” command with custom data and compression factor.</p>
<pre class="dosbatch"><code>$ stm32ai validate -m &lt;model_file_path&gt; -vi test_data.npz -c 8
...
Accuracy report #1 for the generated x86 C-model
------------------------------------------------------------------
NOTE: Computed against the provided ground truth values

acc=94.00%, rmse=0.0224, mae=0.0127

8 classes (50 samples)
------------------------------------------------
C0         4    .    .    .    .    .    .    .
C1         .    8    .    .    .    .    .    .
C2         .    .    6    .    .    .    1    .
C3         .    .    .    7    .    .    .    .
C4         .    .    .    .    5    .    .    .
C5         .    1    .    .    .    6    1    .
C6         .    .    .    .    .    .    5    .
C7         .    .    .    .    .    .    .    6

Accuracy report #1 for the reference model
------------------------------------------------------------------
NOTE: Computed against the provided ground truth values

acc=94.00%, rmse=0.0205, mae=0.0116

8 classes (50 samples)
------------------------------------------------
C0         4    .    .    .    .    .    .    .
C1         .    8    .    .    .    .    .    .
C2         .    .    6    .    .    .    1    .
C3         .    .    .    7    .    .    .    .
C4         .    .    .    .    5    .    .    .
C5         .    1    .    .    .    6    1    .
C6         .    .    .    .    .    .    5    .
C7         .    .    .    .    .    .    .    6

Cross accuracy report #1 (reference vs C-model)
------------------------------------------------------------------
NOTE: the output of the reference model is used as ground truth value

acc=100.00%, rmse=0.0063, mae=0.0034

8 classes (50 samples)
------------------------------------------------
C0         4    .    .    .    .    .    .    .
C1         .    9    .    .    .    .    .    .
C2         .    .    6    .    .    .    .    .
C3         .    .    .    7    .    .    .    .
C4         .    .    .    .    5    .    .    .
C5         .    .    .    .    .    6    .    .
C6         .    .    .    .    .    .    7    .
C7         .    .    .    .    .    .    .    6

Evaluation report (summary)
--------------------------------------------------
Mode                  acc       rmse      mae
--------------------------------------------------
x86 C-model  out=1    94.0%     0.022430  0.012718
original model out=1  94.0%     0.020461  0.011608
X-cross  #1           100.0%    0.006338  0.003373

L2r error : 2.63014436e-02 (expected to be &lt; 0.01)
...</code></pre>
</section>
<section id="ref_q_inputs" class="level2">
<h2>Quantized model - Input samples only</h2>
<p>This case is quite similar to the 32b float model, only the <em>L2r</em> is NOT EVALUATED due to the error of quantization, even if the 32b float format is used for the input/outputs tensors. If the model requests a quantized input tensor, the inputs <code>[I]</code> are previously converted (32b float to fixed-point/integer). Similar operation is applied for the outputs if necessary. For the quantized TFLite model, these operations are not performed if the user provides the int8 or uint8 data samples. As for the 32b float model, [R’] is used as reference to compute the X-cross metrics: <em>ACC</em>, <em>RMSE</em> and <em>MAE</em>.</p>
<div id="fig:id_q_inputs" class="fignos">
<figure>
<img src="" property="center" style="width:85.0%" alt /><figcaption><span>Figure 3:</span> Quantized TFLite model with input samples only</figcaption>
</figure>
</div>
<div class="Warning">
<p><strong>Note</strong> — For the quantized TFLite model, this is a typical case where we can have the same output values between the C-model and the original model: <code>X-cross #1 100%, RMSE=MAE=0.0</code>.</p>
</div>
<div id="fig:id_q_inputs" class="fignos">
<figure>
<img src="" property="center" style="width:85.0%" alt /><figcaption><span>Figure 4:</span> Quantized Keras model with input samples only</figcaption>
</figure>
</div>
<div class="Warning">
<p><strong>Limitation</strong> — int8/uint8 user data set files can be not used for the quantized Keras models.</p>
</div>
<p>Typical output during the execution of the <code>validate</code> command with random inputs for quantized model.</p>
<pre class="dosbatch"><code>$ stm32ai validate -m &lt;modified_model_file&gt;.h5 -q &lt;quant_file_desc&gt;.json
...

Cross accuracy report #1 (reference vs C-model)
----------------------------------------------------------------------------------------------------
NOTE: the output of the reference model is used as ground truth/reference value

 acc=100.00%, rmse=0.006488, mae=0.002825, l2r=0.025700

 10 classes (10 samples)
 ----------------------------------------------------------
 C0        0    .    .    .    .    .    .    .    .    .
 C1        .    0    .    .    .    .    .    .    .    .
 C2        .    .    6    .    .    .    .    .    .    .
 C3        .    .    .    1    .    .    .    .    .    .
 C4        .    .    .    .    1    .    .    .    .    .
 C5        .    .    .    .    .    2    .    .    .    .
 C6        .    .    .    .    .    .    0    .    .    .
 C7        .    .    .    .    .    .    .    0    .    .
 C8        .    .    .    .    .    .    .    .    0    .
 C9        .    .    .    .    .    .    .    .    .    0

Evaluation report (summary)
------------------------------------------------------------------------------------------------
Mode          acc      rmse      mae       l2r       tensor
------------------------------------------------------------------------------------------------
X-cross #1    100.00%  0.006488  0.002825  0.025700  softmax_8 [ai_float, (1, 1, 10), m_id=12]
...</code></pre>
</section>
<section id="tflite-quantized-model---inputoutput-samples" class="level2">
<h2>TFLite Quantized model - Input/output samples</h2>
<p>The input <code>[I]</code> of the model can be optionally converted/quantized before to feed the models. As the reference samples are provided, all metrics for each models are computed including X-cross values for quick evaluation.</p>
<div id="fig:id_q_io" class="fignos">
<figure>
<img src="" property="center" style="width:80.0%" alt /><figcaption><span>Figure 5:</span> Quantized TF lite model with IO values</figcaption>
</figure>
</div>
<p>Typical output during the execution of the <code>&#39;validate&#39;</code> command.</p>
<pre class="dosbatch"><code>$ stm32ai validate -m &lt;modified_model_file&gt;.tflite -vi test_data.npz
...
Accuracy report #1 for the generated x86 C-model
----------------------------------------------------------------------------------------------------
NOTE: Computed against the provided ground truth values

 acc=99.22%, rmse=0.038794, mae=0.002396, l2r=0.123459

 10 classes (128 samples)
 ----------------------------------------------------------
 C0       12    .    .    .    .    .    .    .    .    .
 C1        .   19    .    .    .    .    .    .    .    .
 C2        .    .   16    .    .    .    .    .    .    .
 C3        .    .    .   11    .    .    .    .    .    .
 C4        .    .    .    .   15    .    .    .    .    .
 C5        .    .    .    .    .    7    .    .    .    .
 C6        .    .    .    .    .    .   10    .    .    .
 C7        .    .    .    .    .    .    .    9    .    .
 C8        .    .    .    .    .    .    .    .   18    .
 C9        .    .    .    .    .    1    .    .    .   10

Accuracy report #1 for the reference model
----------------------------------------------------------------------------------------------------
NOTE: Computed against the provided ground truth values

 acc=99.22%, rmse=0.038794, mae=0.002396, l2r=0.123459

 10 classes (128 samples)
 ----------------------------------------------------------
 C0       12    .    .    .    .    .    .    .    .    .
 C1        .   19    .    .    .    .    .    .    .    .
 C2        .    .   16    .    .    .    .    .    .    .
 C3        .    .    .   11    .    .    .    .    .    .
 C4        .    .    .    .   15    .    .    .    .    .
 C5        .    .    .    .    .    7    .    .    .    .
 C6        .    .    .    .    .    .   10    .    .    .
 C7        .    .    .    .    .    .    .    9    .    .
 C8        .    .    .    .    .    .    .    .   18    .
 C9        .    .    .    .    .    1    .    .    .   10

Cross accuracy report #1 (reference vs C-model)
----------------------------------------------------------------------------------------------------
NOTE: the output of the reference model is used as ground truth/reference value

 acc=100.00%, rmse=0.000000, mae=0.000000, l2r=0.000000

 10 classes (128 samples)
 ----------------------------------------------------------
 C0       12    .    .    .    .    .    .    .    .    .
 C1        .   19    .    .    .    .    .    .    .    .
 C2        .    .   16    .    .    .    .    .    .    .
 C3        .    .    .   11    .    .    .    .    .    .
 C4        .    .    .    .   15    .    .    .    .    .
 C5        .    .    .    .    .    8    .    .    .    .
 C6        .    .    .    .    .    .   10    .    .    .
 C7        .    .    .    .    .    .    .    9    .    .
 C8        .    .    .    .    .    .    .    .   18    .
 C9        .    .    .    .    .    .    .    .    .   10


Evaluation report (summary)
--------------------------------------------------------------------------------------------------
Mode                 acc      rmse      mae       l2r       tensor
--------------------------------------------------------------------------------------------------
x86 C-model #1       99.22%   0.038794  0.002396  0.123459  nl_5_fmt [ai_i8, (1, 1, 10), m_id=5]
original model #1    99.22%   0.038794  0.002396  0.123459  nl_5_fmt [ai_i8, (1, 1, 10), m_id=5]
X-cross #1           100.00%  0.000000  0.000000  0.000000  nl_5_fmt [ai_i8, (1, 1, 10), m_id=5]
...</code></pre>
</section>
<section id="model-with-multiple-io" class="level2">
<h2>Model with multiple IO</h2>
<p>All metrics are calculated independently for each outputs including the <code>L2r</code> for the 32b float model.</p>
<pre class="dosbatch"><code>$ stm32ai validate -m &lt;model_with_2_outputs&gt;
...
Cross accuracy report #1 (reference vs C-model)
------------------------------------------------------------------
NOTE: the output of the reference model is used as ground truth value

acc=100.00%, rmse=0.0000, mae=0.0000

Cross accuracy report #2 (reference vs C-model)
------------------------------------------------------------------
NOTE: the output of the reference model is used as ground truth value

acc=100.00%, rmse=0.0000, mae=0.0000
...
Evaluation report (summary)
-------------------------------------------------------------------------------------------------
Mode          acc      rmse      mae       l2r       tensor
-------------------------------------------------------------------------------------------------
X-cross #1    n.a.     0.000000  0.000000  0.000000  add_1 [ai_float, (16, 16, 3), m_id=7]
X-cross #2    n.a.     0.000000  0.000000  0.000000  multiply_1 [ai_float, (16, 16, 3), m_id=7]

L2r error : 0.00000000e+00 (expected to be &lt; 0.01)
...</code></pre>
<pre class="dosbatch"><code>$ stm32ai validate -m &lt;model_with_2_outputs&gt; -vi test_multiple_io.npz
...
Evaluation report (summary)
--------------------------------------------------------------------------------------------------------
Mode                 acc      rmse      mae       l2r       tensor
--------------------------------------------------------------------------------------------------------
x86 C-model #1       n.a.     0.000000  0.000000  0.000000  add_1 [ai_float, (16, 16, 3), m_id=7]
x86 C-model #2       n.a.     0.000000  0.000000  0.000000  multiply_1 [ai_float, (16, 16, 3), m_id=7]
original model #1    n.a.     0.000000  0.000000  0.000000  add_1 [ai_float, (16, 16, 3), m_id=7]
original model #2    n.a.     0.000000  0.000000  0.000000  multiply_1 [ai_float, (16, 16, 3), m_id=7]
X-cross #1           n.a.     0.000000  0.000000  0.000000  add_1 [ai_float, (16, 16, 3), m_id=7]
X-cross #2           n.a.     0.000000  0.000000  0.000000  multiply_1 [ai_float, (16, 16, 3), m_id=7]

L2r error : 0.00000000e+00 (expected to be &lt; 0.01)</code></pre>
<pre class="dosbatch"><code>Complexity/l2r error per-layer - macc=4,608 rom=0
----------------------------------------------------------------------------------------------------
id  layer (type)         macc                      rom                   l2r error
----------------------------------------------------------------------------------------------------
3   maximum_1 (Eltwise)  |||||||||||||||||  33.3%  |                0.0%
4   minimum_1 (Eltwise)  |||||||            16.7%  |                0.0%
5   add_1 (Eltwise)      |||||||            16.7%  |                0.0%  0.00000000e+00 *
6   subtract_1 (Eltwise) |||||||            16.7%  |                0.0%
7   multiply_1 (Eltwise) |||||||            16.7%  |                0.0%  0.00000000e+00 *
----------------------------------------------------------------------------------------------------</code></pre>
</section>
</section>
<section id="ref_script_ex" class="level1">
<h1>Post-processing example</h1>
<p>Following log is the output of a custom Python script to read the generated samples and to build new element-wise metrics: <code>&#39;variance&#39;</code> and <code>&#39;f1_score&#39;</code> (thanks to the <em>numpy</em> and <em>sklearn.metrics</em> Python modules). <code>&#39;acc&#39;</code>, <code>&#39;rmse&#39;</code>, <code>&#39;mae&#39;</code> and <code>&#39;l2r&#39;</code> are also provided to illustrate how these metrics are computed.</p>
<pre class="dosbatch"><code>$ python custom_metrics.py
Read generated NPZ file &quot;./stm32ai_output\network_val_io.npz&quot;...
Read generated CSV files &quot;./stm32ai_output\network_val_*.csv&quot;...
Read reference NPZ file &quot;network_reference.npz&quot;...

Evaluation report
--------------------------------------------------------------------------------
               acc       rmse      mae       var       f1_score  l2r
--------------------------------------------------------------------------------
C-model        94.0%     0.149257  0.082441  0.022333  0.932265  0.44756502
original model 94.0%     0.020461  0.011608  0.000420  0.944341  0.08464438
X-cross        92.0%     0.156579  0.087279  0.024577  0.919234  0.46952036</code></pre>
<p>Full code of the custom Python script.</p>
<div class="sourceCode" id="cb23"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb23-1"><a href="#cb23-1"></a><span class="co"># -*- coding: utf-8 -*-</span></span>
<span id="cb23-2"><a href="#cb23-2"></a><span class="co">&quot;&quot;&quot;</span></span>
<span id="cb23-3"><a href="#cb23-3"></a><span class="co">Implement custom metrics</span></span>
<span id="cb23-4"><a href="#cb23-4"></a><span class="co">&quot;&quot;&quot;</span></span>
<span id="cb23-5"><a href="#cb23-5"></a><span class="im">from</span> __future__ <span class="im">import</span> absolute_import</span>
<span id="cb23-6"><a href="#cb23-6"></a><span class="im">from</span> __future__ <span class="im">import</span> division</span>
<span id="cb23-7"><a href="#cb23-7"></a><span class="im">from</span> __future__ <span class="im">import</span> print_function</span>
<span id="cb23-8"><a href="#cb23-8"></a></span>
<span id="cb23-9"><a href="#cb23-9"></a><span class="im">import</span> os</span>
<span id="cb23-10"><a href="#cb23-10"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb23-11"><a href="#cb23-11"></a><span class="im">import</span> argparse</span>
<span id="cb23-12"><a href="#cb23-12"></a></span>
<span id="cb23-13"><a href="#cb23-13"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> mean_squared_error</span>
<span id="cb23-14"><a href="#cb23-14"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> accuracy_score</span>
<span id="cb23-15"><a href="#cb23-15"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> f1_score</span>
<span id="cb23-16"><a href="#cb23-16"></a></span>
<span id="cb23-17"><a href="#cb23-17"></a>OUTPUT_DIR <span class="op">=</span> <span class="st">&#39;./stm32ai_output&#39;</span></span>
<span id="cb23-18"><a href="#cb23-18"></a>NETWORK_NAME <span class="op">=</span> <span class="st">&#39;network&#39;</span></span>
<span id="cb23-19"><a href="#cb23-19"></a>REFERENCE_NPZ <span class="op">=</span> <span class="st">&#39;network_reference.npz&#39;</span></span>
<span id="cb23-20"><a href="#cb23-20"></a></span>
<span id="cb23-21"><a href="#cb23-21"></a><span class="co"># metrics</span></span>
<span id="cb23-22"><a href="#cb23-22"></a></span>
<span id="cb23-23"><a href="#cb23-23"></a><span class="kw">def</span> mse(ref, pred):</span>
<span id="cb23-24"><a href="#cb23-24"></a>  <span class="co">&quot;&quot;&quot;Return Mean Squared Error (MSE).&quot;&quot;&quot;</span></span>
<span id="cb23-25"><a href="#cb23-25"></a>  <span class="cf">return</span> ((ref <span class="op">-</span> pred).astype(np.float64) <span class="op">**</span> <span class="dv">2</span>).mean()</span>
<span id="cb23-26"><a href="#cb23-26"></a>  </span>
<span id="cb23-27"><a href="#cb23-27"></a><span class="kw">def</span> rmse(ref, pred):</span>
<span id="cb23-28"><a href="#cb23-28"></a>  <span class="co">&quot;&quot;&quot;Return Root Mean Squared Error (RMSE).&quot;&quot;&quot;</span></span>
<span id="cb23-29"><a href="#cb23-29"></a>  <span class="cf">return</span> np.sqrt(((ref <span class="op">-</span> pred).astype(np.float64) <span class="op">**</span> <span class="dv">2</span>).mean())</span>
<span id="cb23-30"><a href="#cb23-30"></a>  </span>
<span id="cb23-31"><a href="#cb23-31"></a><span class="kw">def</span> mae(ref, pred):</span>
<span id="cb23-32"><a href="#cb23-32"></a>  <span class="co">&quot;&quot;&quot;Return Mean Absolute Error (MAE).&quot;&quot;&quot;</span></span>
<span id="cb23-33"><a href="#cb23-33"></a>  <span class="cf">return</span> (np.<span class="bu">abs</span>(ref <span class="op">-</span> pred).astype(np.float64)).mean()</span>
<span id="cb23-34"><a href="#cb23-34"></a></span>
<span id="cb23-35"><a href="#cb23-35"></a><span class="kw">def</span> var(ref, pred):</span>
<span id="cb23-36"><a href="#cb23-36"></a>  <span class="co">&quot;&quot;&quot;Return Variance&quot;&quot;&quot;</span></span>
<span id="cb23-37"><a href="#cb23-37"></a>  <span class="cf">return</span> np.var((ref <span class="op">-</span> pred), dtype<span class="op">=</span>np.float64, ddof<span class="op">=</span><span class="dv">1</span>)</span>
<span id="cb23-38"><a href="#cb23-38"></a></span>
<span id="cb23-39"><a href="#cb23-39"></a><span class="kw">def</span> acc(ref, pred):</span>
<span id="cb23-40"><a href="#cb23-40"></a>  <span class="co">&quot;&quot;&quot;Classification accuracy (ACC).&quot;&quot;&quot;</span></span>
<span id="cb23-41"><a href="#cb23-41"></a>  <span class="cf">return</span> accuracy_score(np.argmax(ref, axis<span class="op">=</span><span class="dv">1</span>), np.argmax(pred, axis<span class="op">=</span><span class="dv">1</span>))</span>
<span id="cb23-42"><a href="#cb23-42"></a></span>
<span id="cb23-43"><a href="#cb23-43"></a><span class="kw">def</span> f1_s(ref, pred, average<span class="op">=</span><span class="st">&#39;macro&#39;</span>):</span>
<span id="cb23-44"><a href="#cb23-44"></a>  <span class="co">&quot;&quot;&quot;Compute the F1 score, also known as balanced F-score or F-measure (F1)&quot;&quot;&quot;</span></span>
<span id="cb23-45"><a href="#cb23-45"></a>  <span class="cf">return</span> f1_score(np.argmax(ref, axis<span class="op">=</span><span class="dv">1</span>), np.argmax(pred, axis<span class="op">=</span><span class="dv">1</span>), average<span class="op">=</span>average)</span>
<span id="cb23-46"><a href="#cb23-46"></a></span>
<span id="cb23-47"><a href="#cb23-47"></a><span class="kw">def</span> l2r(ref, pred):</span>
<span id="cb23-48"><a href="#cb23-48"></a>  <span class="co">&quot;&quot;&quot;Compute L2 relative error&quot;&quot;&quot;</span></span>
<span id="cb23-49"><a href="#cb23-49"></a>  <span class="kw">def</span> magnitude(v):</span>
<span id="cb23-50"><a href="#cb23-50"></a>    <span class="cf">return</span> np.sqrt(np.<span class="bu">sum</span>(np.square(v).flatten()))</span>
<span id="cb23-51"><a href="#cb23-51"></a>  mag <span class="op">=</span> magnitude(pred) <span class="op">+</span> np.finfo(np.float32).eps</span>
<span id="cb23-52"><a href="#cb23-52"></a>  <span class="cf">return</span> magnitude(ref <span class="op">-</span> pred) <span class="op">/</span> mag</span>
<span id="cb23-53"><a href="#cb23-53"></a></span>
<span id="cb23-54"><a href="#cb23-54"></a><span class="co"># read the generated inputs and predicted samples (origninal &amp; C models)</span></span>
<span id="cb23-55"><a href="#cb23-55"></a></span>
<span id="cb23-56"><a href="#cb23-56"></a>fname <span class="op">=</span> os.path.join(OUTPUT_DIR, NETWORK_NAME <span class="op">+</span> <span class="st">&#39;_val_io.npz&#39;</span>)</span>
<span id="cb23-57"><a href="#cb23-57"></a><span class="bu">print</span>(<span class="st">&#39;Read generated NPZ file &quot;</span><span class="sc">{}</span><span class="st">&quot;...&#39;</span>.<span class="bu">format</span>(fname))</span>
<span id="cb23-58"><a href="#cb23-58"></a>arrays <span class="op">=</span> np.load(fname)</span>
<span id="cb23-59"><a href="#cb23-59"></a></span>
<span id="cb23-60"><a href="#cb23-60"></a>i_  <span class="op">=</span> arrays[<span class="st">&#39;m_inputs_1&#39;</span>]</span>
<span id="cb23-61"><a href="#cb23-61"></a>ic_  <span class="op">=</span> arrays[<span class="st">&#39;c_inputs_1&#39;</span>]</span>
<span id="cb23-62"><a href="#cb23-62"></a>p_  <span class="op">=</span> arrays[<span class="st">&#39;c_outputs_1&#39;</span>]</span>
<span id="cb23-63"><a href="#cb23-63"></a>pm_ <span class="op">=</span> arrays[<span class="st">&#39;m_outputs_1&#39;</span>]</span>
<span id="cb23-64"><a href="#cb23-64"></a></span>
<span id="cb23-65"><a href="#cb23-65"></a>fname <span class="op">=</span> os.path.join(OUTPUT_DIR, NETWORK_NAME <span class="op">+</span> <span class="st">&#39;_val_*.csv&#39;</span>)</span>
<span id="cb23-66"><a href="#cb23-66"></a><span class="bu">print</span>(<span class="st">&#39;Read generated CSV files &quot;</span><span class="sc">{}</span><span class="st">&quot;...&#39;</span>.<span class="bu">format</span>(fname))</span>
<span id="cb23-67"><a href="#cb23-67"></a></span>
<span id="cb23-68"><a href="#cb23-68"></a>fname <span class="op">=</span> os.path.join(OUTPUT_DIR, NETWORK_NAME <span class="op">+</span> <span class="st">&#39;_val_m_inputs_1.csv&#39;</span>)</span>
<span id="cb23-69"><a href="#cb23-69"></a>i_csv_ <span class="op">=</span> np.atleast_2d(np.genfromtxt(fname, delimiter<span class="op">=</span><span class="st">&#39;,&#39;</span>))</span>
<span id="cb23-70"><a href="#cb23-70"></a>fname <span class="op">=</span> os.path.join(OUTPUT_DIR, NETWORK_NAME <span class="op">+</span> <span class="st">&#39;_val_c_inputs_1.csv&#39;</span>)</span>
<span id="cb23-71"><a href="#cb23-71"></a>ic_csv_ <span class="op">=</span> np.atleast_2d(np.genfromtxt(fname, delimiter<span class="op">=</span><span class="st">&#39;,&#39;</span>))</span>
<span id="cb23-72"><a href="#cb23-72"></a>fname <span class="op">=</span> os.path.join(OUTPUT_DIR, NETWORK_NAME <span class="op">+</span> <span class="st">&#39;_val_c_outputs_1.csv&#39;</span>)</span>
<span id="cb23-73"><a href="#cb23-73"></a>p_csv_ <span class="op">=</span> np.atleast_2d(np.genfromtxt(fname, delimiter<span class="op">=</span><span class="st">&#39;,&#39;</span>))</span>
<span id="cb23-74"><a href="#cb23-74"></a>fname <span class="op">=</span> os.path.join(OUTPUT_DIR, NETWORK_NAME <span class="op">+</span> <span class="st">&#39;_val_m_outputs_1.csv&#39;</span>)</span>
<span id="cb23-75"><a href="#cb23-75"></a>pm_csv_ <span class="op">=</span> np.atleast_2d(np.genfromtxt(fname, delimiter<span class="op">=</span><span class="st">&#39;,&#39;</span>))</span>
<span id="cb23-76"><a href="#cb23-76"></a></span>
<span id="cb23-77"><a href="#cb23-77"></a>l_csv <span class="op">=</span> <span class="bu">len</span>(i_csv_)</span>
<span id="cb23-78"><a href="#cb23-78"></a></span>
<span id="cb23-79"><a href="#cb23-79"></a><span class="co"># read reference samples</span></span>
<span id="cb23-80"><a href="#cb23-80"></a></span>
<span id="cb23-81"><a href="#cb23-81"></a><span class="bu">print</span>(<span class="st">&#39;Read reference NPZ file &quot;</span><span class="sc">{}</span><span class="st">&quot;...&#39;</span>.<span class="bu">format</span>(REFERENCE_NPZ))</span>
<span id="cb23-82"><a href="#cb23-82"></a></span>
<span id="cb23-83"><a href="#cb23-83"></a>fname <span class="op">=</span> REFERENCE_NPZ</span>
<span id="cb23-84"><a href="#cb23-84"></a>arrays <span class="op">=</span> np.load(fname)</span>
<span id="cb23-85"><a href="#cb23-85"></a></span>
<span id="cb23-86"><a href="#cb23-86"></a>i_ref <span class="op">=</span> arrays[<span class="st">&#39;in_0&#39;</span>]</span>
<span id="cb23-87"><a href="#cb23-87"></a>r_ref <span class="op">=</span> arrays[<span class="st">&#39;out_0&#39;</span>]</span>
<span id="cb23-88"><a href="#cb23-88"></a>  </span>
<span id="cb23-89"><a href="#cb23-89"></a><span class="co"># reshape the _csv data</span></span>
<span id="cb23-90"><a href="#cb23-90"></a></span>
<span id="cb23-91"><a href="#cb23-91"></a>i_csv_ <span class="op">=</span> i_csv_.reshape((<span class="op">-</span><span class="dv">1</span>,) <span class="op">+</span> i_.shape[<span class="dv">1</span>:])</span>
<span id="cb23-92"><a href="#cb23-92"></a>p_csv_ <span class="op">=</span> p_csv_.reshape((<span class="op">-</span><span class="dv">1</span>,) <span class="op">+</span> p_.shape[<span class="dv">1</span>:])</span>
<span id="cb23-93"><a href="#cb23-93"></a>pm_csv_ <span class="op">=</span> pm_csv_.reshape((<span class="op">-</span><span class="dv">1</span>,) <span class="op">+</span> pm_.shape[<span class="dv">1</span>:])</span>
<span id="cb23-94"><a href="#cb23-94"></a></span>
<span id="cb23-95"><a href="#cb23-95"></a><span class="co"># calculate metrics</span></span>
<span id="cb23-96"><a href="#cb23-96"></a></span>
<span id="cb23-97"><a href="#cb23-97"></a><span class="kw">def</span> build_metrics(ref, pred):</span>
<span id="cb23-98"><a href="#cb23-98"></a>  res <span class="op">=</span> {}</span>
<span id="cb23-99"><a href="#cb23-99"></a>  res[<span class="st">&#39;acc&#39;</span>] <span class="op">=</span> acc(ref, pred)</span>
<span id="cb23-100"><a href="#cb23-100"></a>  res[<span class="st">&#39;var&#39;</span>] <span class="op">=</span> var(ref, pred)</span>
<span id="cb23-101"><a href="#cb23-101"></a>  res[<span class="st">&#39;f1_score&#39;</span>] <span class="op">=</span> f1_s(ref, pred)</span>
<span id="cb23-102"><a href="#cb23-102"></a>  res[<span class="st">&#39;rmse&#39;</span>] <span class="op">=</span> rmse(ref, pred)</span>
<span id="cb23-103"><a href="#cb23-103"></a>  res[<span class="st">&#39;mae&#39;</span>] <span class="op">=</span> mae(ref, pred)</span>
<span id="cb23-104"><a href="#cb23-104"></a>  res[<span class="st">&#39;mse&#39;</span>] <span class="op">=</span> mse(ref, pred)</span>
<span id="cb23-105"><a href="#cb23-105"></a>  res[<span class="st">&#39;l2r&#39;</span>] <span class="op">=</span> l2r(ref, pred)</span>
<span id="cb23-106"><a href="#cb23-106"></a>  <span class="cf">return</span> res</span>
<span id="cb23-107"><a href="#cb23-107"></a></span>
<span id="cb23-108"><a href="#cb23-108"></a><span class="kw">def</span> print_metrics(name, ref, pred):</span>
<span id="cb23-109"><a href="#cb23-109"></a>  res <span class="op">=</span> build_metrics(ref, pred)</span>
<span id="cb23-110"><a href="#cb23-110"></a>  <span class="bu">str</span> <span class="op">=</span> <span class="st">&#39;</span><span class="sc">{:15s}</span><span class="st">&#39;</span>.<span class="bu">format</span>(name)</span>
<span id="cb23-111"><a href="#cb23-111"></a>  _acc <span class="op">=</span> <span class="st">&#39;</span><span class="sc">{:.1f}</span><span class="st">%&#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;acc&#39;</span>] <span class="op">*</span> <span class="fl">100.0</span>)</span>
<span id="cb23-112"><a href="#cb23-112"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:10s}</span><span class="st">&#39;</span>.<span class="bu">format</span>(_acc)</span>
<span id="cb23-113"><a href="#cb23-113"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.6f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;rmse&#39;</span>])</span>
<span id="cb23-114"><a href="#cb23-114"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.6f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;mae&#39;</span>])</span>
<span id="cb23-115"><a href="#cb23-115"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.6f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;var&#39;</span>])</span>
<span id="cb23-116"><a href="#cb23-116"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.6f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;f1_score&#39;</span>])</span>
<span id="cb23-117"><a href="#cb23-117"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.8f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;l2r&#39;</span>])</span>
<span id="cb23-118"><a href="#cb23-118"></a>  <span class="bu">print</span>(<span class="bu">str</span>)</span>
<span id="cb23-119"><a href="#cb23-119"></a></span>
<span id="cb23-120"><a href="#cb23-120"></a><span class="bu">print</span>(<span class="st">&#39;</span><span class="ch">\n</span><span class="st">Evaluation report&#39;</span>)</span>
<span id="cb23-121"><a href="#cb23-121"></a><span class="bu">print</span>(<span class="st">&#39;-&#39;</span><span class="op">*</span><span class="dv">80</span>)</span>
<span id="cb23-122"><a href="#cb23-122"></a><span class="bu">print</span>(<span class="st">&#39;               </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">&#39;</span>.<span class="bu">format</span>(<span class="st">&#39;acc&#39;</span>, <span class="st">&#39;rmse&#39;</span>,</span>
<span id="cb23-123"><a href="#cb23-123"></a>          <span class="st">&#39;mae&#39;</span>, <span class="st">&#39;var&#39;</span>, <span class="st">&#39;f1_score&#39;</span>, <span class="st">&#39;l2r&#39;</span>))</span>
<span id="cb23-124"><a href="#cb23-124"></a><span class="bu">print</span>(<span class="st">&#39;-&#39;</span><span class="op">*</span><span class="dv">80</span>)</span>
<span id="cb23-125"><a href="#cb23-125"></a>print_metrics(<span class="st">&#39;C-model&#39;</span>, r_ref, p_)</span>
<span id="cb23-126"><a href="#cb23-126"></a>print_metrics(<span class="st">&#39;original model&#39;</span>, r_ref, pm_)</span>
<span id="cb23-127"><a href="#cb23-127"></a>print_metrics(<span class="st">&#39;X-cross&#39;</span>, pm_, p_)</span></code></pre></div>
</section>
<section id="references" class="level1">
<h1>References</h1>
<table style="width:92%;">
<colgroup>
<col style="width: 13%"></col>
<col style="width: 77%"></col>
</colgroup>
<tbody>
<tr class="odd">
<td style="text-align: left;">[1]</td>
<td style="text-align: left;">X-CUBE-AI - <em>AI expansion pack for STM32CubeMX</em><br />
<a href="https://www.st.com/en/embedded-software/x-cube-ai.html">https://www.st.com/en/embedded-software/x-cube-ai.html</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[2]</td>
<td style="text-align: left;">User manual - Getting started with X-CUBE-AI Expansion Package for Artificial Intelligence (AI) <a href="https://www.st.com/resource/en/user_manual/dm00570145.pdf">(pdf)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[3]</td>
<td style="text-align: left;">stm32ai - Command Line Interface <a href="command_line_interface.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[4]</td>
<td style="text-align: left;">Supported Deep Learning toolboxes and layers <a href="layer-support.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[5]</td>
<td style="text-align: left;">Embedded inference client API <a href="embedded_client_api.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[6]</td>
<td style="text-align: left;">Evaluation report and metrics <a href="evaluation_metrics.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[7]</td>
<td style="text-align: left;">FAQs <a href="faqs.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[8]</td>
<td style="text-align: left;">Quantization and quantize command <a href="quantization.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[9]</td>
<td style="text-align: left;">Relocatable binary network support <a href="relocatable.html">(link)</a></td>
</tr>
</tbody>
</table>
</section>
<section id="revision-history" class="level1">
<h1>Revision history</h1>
<table>
<colgroup>
<col style="width: 32%"></col>
<col style="width: 24%"></col>
<col style="width: 44%"></col>
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">Date</th>
<th style="text-align: left;">version</th>
<th style="text-align: left;">changes</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;"><strong>2019-07-02</strong></td>
<td style="text-align: left;">r1.0</td>
<td style="text-align: left;">initial version</td>
</tr>
<tr class="even">
<td style="text-align: left;"><strong>2019-09-20</strong></td>
<td style="text-align: left;">r1.1</td>
<td style="text-align: left;">X-CUBE-AI 4.1 update,  add MACC/ROM/RAM metrics + Quantized TF lite model.</td>
</tr>
<tr class="odd">
<td style="text-align: left;"><strong>2019-12-06</strong></td>
<td style="text-align: left;">r1.2</td>
<td style="text-align: left;">X-CUBE-AI 5.0 update,  add network with multiple IO.</td>
</tr>
<tr class="even">
<td style="text-align: left;"><strong>2020-05-12</strong></td>
<td style="text-align: left;">r1.3</td>
<td style="text-align: left;">X-CUBE-AI 5.1 update,  add int8/uint8 support for input validation data files,  complete the description of the metrics</td>
</tr>
<tr class="odd">
<td style="text-align: left;"><strong>2020-09-15</strong></td>
<td style="text-align: left;">r1.4</td>
<td style="text-align: left;">X-CUBE-AI 5.2 update, L2r metric is always calculated. Name/shape of the output tensor is added with the results</td>
</tr>
</tbody>
</table>
</section>



<section class="st_footer">

<h1> <br> </h1>

<p style="font-family:verdana; text-align:left;">
 Embedded Documentation 

	- <b> Evaluation report and metrics </b>
			<br> X-CUBE-AI Expansion Package
				<br> r1.4
		 - AI PLATFORM r5.2.0
			 (Embedded Inference Client API 1.1.0) 
			 - Command Line Interface r1.4.0 
		
	
</p>

<img src="" title="ST logo" align="right" height="100" />

<div class="stnotice">
Information in this document is provided solely in connection with ST products.
The contents of this document are subject to change without prior notice.
<br>
© Copyright STMicroelectronics 2020. All rights reserved. <a href="http://www.st.com">www.st.com</a>
</div>

<hr size="1" />
</section>


</article>
</body>

</html>
